Repository navigation
End spans that a plugin leaks after its interceptor returns or BeforeInvoke fails, so tracing continues - #266
Merged
Conversation
…Invoke fails, so tracing continues
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
7 open issues
InvokeInterceptedcurrently proceeds to run the interceptedcall()and theAfterInvokedefer… · NewInvokeInterceptedcurrently proceeds to run the interceptedcall()and theAfterInvokedefer… · NewBeforeInvokeerror/panic paths don't callinvocation.markExited(). If a plugin calls… · NewBeforeInvokeerror/panic paths don't callinvocation.markExited(). If a plugin calls… · New The PR description says the agent 'logs a warning for each' reclaimed leaked span.endLeakedSpan… · New The PR description says the agent 'logs a warning for each' reclaimed leaked span.endLeakedSpan… · NewafterInnerDepthis computed fromouterrather thanafterInner, so the test doesn't actually… · New
What changed in this PR
This PR addresses leaked spans in Go agent interceptors so tracing doesn’t get “stuck” (especially with noop spans) after buggy plugins or failed BeforeInvoke hooks.
Changes:
- Add span ownership + orphan reclamation/rollback mechanisms to prevent leaked spans from affecting later requests on the same goroutine.
- Update code generation templates to mark interceptor exit and to rollback spans created by failed/panicking
BeforeInvoke. - Fix rewriter builtin handling and add extensive unit tests + documentation for span lifecycle rules.
| File | Description |
|---|---|
| tools/go-agent/instrument/plugins/templates/method_intercept_before.tmpl | Marks/rolls back active span around BeforeInvoke and marks invocation exited on skip path. |
| tools/go-agent/instrument/plugins/templates/method_intercept_after.tmpl | Ensures invocations are marked exited even when AfterInvoke panics. |
| tools/go-agent/instrument/plugins/rewrite/context.go | Prevents renaming of all Go predeclared builtin functions during rewrite. |
| tools/go-agent/instrument/plugins/rewrite/context_test.go | Adds test coverage ensuring builtins are not renamed in rewritten agent-core copies. |
| tools/go-agent/instrument/plugins/enhance_method_test.go | Validates generated before/after template code ordering (mark, rollback, and markExited). |
| plugins/grpc-v184/client_recvmsg_interceptor.go | Wraps RecvMsg span in a holder so it can intentionally remain active until trailer read. |
| plugins/core/tracing/bridge.go | Adds span-owner binding hook (BindOwner) at the tracing bridge wrapper layer. |
| plugins/core/tracing/bridge_test.go | Tests that SpanWrapper.BindOwner forwards ownership when supported by the wrapped span. |
| plugins/core/tracing.go | Ensures orphan spans are ended before creating a new span under an active context. |
| plugins/core/span_owner.go | Introduces orphan detection, reclaim, and rollback logic; adds owner binding for spans/noops. |
| plugins/core/span_owner_test.go | Adds extensive tests for rollback/reclaim behavior across ignored/noop/continued/reused spans. |
| plugins/core/span_noop.go | Tracks noop span ownership + continued-snapshot marker for reclaim/rollback rules. |
| plugins/core/span_default.go | Tracks reuse owners and ownership-stored status to keep reused entry spans active appropriately. |
| plugins/core/operator/invocation.go | Binds span ownership to invocations via SetContext; adds markExited and rollback helpers. |
| plugins/core/operator/invocation_test.go | Tests span ownership binding and rollback helper behavior. |
| plugins/core/operator/intercepter_test_base.go | Adds a helper to simulate generated interception order for tests. |
| docs/en/development-and-contribution/development-guide.md | Documents interceptor span lifecycle rules and how reclaim/rollback works. |
| CHANGES.md | Release notes for span leak reclamation, reused-entry behavior, and builtin rewrite fix. |
Review effort: Lite
Comment on lines
+139
to
+143
| if t.Log != nil { | ||
| t.Log.Warnf("ending the span %q that its plugin did not end", name) | ||
| } | ||
| span.End() | ||
| } |
Copilot stopped reviewing on behalf of
mrproliu due to an error
November 9, 2026 14:30
wu-sheng
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Follow-up to #265.
Motivation
Since #265, a request that is ignored, or that runs while the agent is disconnected, stores a noop span as the active span. If a plugin leaks that span, every later span on the same goroutine is also a noop. With the default config, that goroutine then stops tracing for good. A plugin can leak a span in two ways:
BeforeInvokestores the span withSetContext, butAfterInvokenever ends it.BeforeInvokecreates a span, then returns an error or panics before callingSetContext.Leaked real spans have a similar effect: they stay on the stack and become the parent of unrelated spans.
Changes
SetContextbinds the stored span to the current interception, and the generated after-code marks that interception as exited. The next time a span is created, orActiveSpanMarkis called, the core ends every active span whose interception has already exited. It logs a warning for each one and counts it insw_go_possible_leaked_context_counter. One reclaim pass ends at most 64 spans. A continued snapshot is never ended.BeforeInvokeis rolled back. The before-template records the depth of the active span stack before callingBeforeInvoke. IfBeforeInvokereturns an error or panics without callingSetContext, the spans it created are ended. A context that the hook continued withContinueContextis kept.SetContext, ending the nested call no longer pops the span. It stays active until the outer request ends. Callers that only create and end the span keep their current behavior, for example kratos middleware, consumers that pull messages inside a request, and the toolkit.RecvMsgkeeps its span active on purpose until the trailer is read. It now stores the span in a holder, so the span is not reclaimed whenRecvMsgreturns.closein the copied operator code was renamed to a package function, which broke instrumented builds.CHANGES.mdentries.Tests
make testandmake test-racepass. golangci-lint v2.15.0 reports no issues.main:kratosv2,gin,http,grpc,gorm,so11y,cross-goroutine, and the rest pass with unchanged expected data.pulsarpasses when it runs alone.segmentio-kafkahas a startup race between the app and Kafka.mainshows the same race.pproffails onmaintoo. It expectshttp.params, but the scenario doesn't enable parameter collection.